Image retrieving apparatus

ABSTRACT

In an image retrieving apparatus for retrieving a retrieving image in an input image, a color histogram of an image in a retrieving area in the input image is compared with a color histogram of the retrieving image. At first, a candidate area in which the retrieving image can be included is roughly retrieved by rough image retrieving with selecting a larger retrieving area and a rough resolution of gradation of the histograms. Subsequently, an area including an image corresponding to the retrieving image is precisely retrieved by fine image retrieving with a smaller retrieving area and a fine resolution of gradation of the histograms.

This application is based on patent applications 2000-339306 and2000-361566 filed in Japan, the contents of which are herebyincorporated by references.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image retrieving technique forjudging whether an image in an area of an input image is similar to orthe same as a predetermined reference image or not.

2. Description of the Related Art

Color histogram is conventionally used for judging whether an image inan area of an input image is similar to or the same as a predeterminedreference image or not. In the method using the color histogram, a colorhistogram of the reference image and a color histogram of an image in apredetermined area in the input image are compared. The area in theinput image to be compared is moved at a predetermined pitch in thehorizontal and vertical directions on the whole input image. Theidentity or the similarity of the color histogram of each portion of theinput image and the color histogram of the reference image arecalculated. The area in the input image having the identity or thelargest similarity is judged as the area the same as or similar to thereference image. The size of the area in the input image to be comparedcan be varied corresponding to the size of the reference image.

For increasing the processing speed of the image retrieving by themethod using the color histogram, it is proposed to vary the pitch ofthe movement of the area to be compared corresponding to the similarityof the color histograms (see collection of congress of electronicinformation and communication D-II Vol.J81-D-II No.9 pp.2035-2042September 1998). In this modification, when the area to be compared isin the vicinity of the area having the lower similarity, the pitch ofthe movement of the area is varied to be larger. When the area to becompared is in the vicinity of the area having the higher similarity,the pitch of the movement of the area is varied to be smaller. As aresult, the processing speed of the image reference can be made faster.

In the above-mentioned conventional methods, the color histogram iscalculated with respect to each image in the area and compared with thatof the reference image. Furthermore, the size of the area to be comparedcan be varied corresponding to the size of the reference image, so thatthe burden of the image processing becomes larger. Thus, a highperformance computer is necessary for processing the image reference.Furthermore, when the color histogram having a fine resolution ofgradation is used, the quantity of the calculation necessary forreferencing the color histograms becomes much larger.

Actually, it is desired to know whether a predetermined kind of imagesuch as a person is included in the input image or not, instead ofjudging whether the same image as the reference image is included in theinput image or not. In such the case, the size of the area to becompared is generally known. Thus, it is desired to propose a new methodfor judging whether a predetermined kind of image is included in theinput image or not by a calculation performance such as a one-chipmicrocomputer used in a household electric appliance.

On the other hand, in the image retrieving of the input image by usingthe color histogram, the number of image data of the input image or thereference image is sometimes small, when the density of the image issmall or when the size of the image to be compared is small. In such thecase, the color histogram will take a comb shape or a discrete histogramincluding the gradation of zero degree.

An example that both of the numbers of the image data of the input imageand the reference image are small is described with reference to FIGS.30A to 30E. FIG. 30A shows an input image 201. FIG. 30B shows areference image 202. Numeral 203 in FIG. 30A designates an area to beretrieved. FIG. 30C shows a normalized color histogram of the area 203.FIG. 30D shows a normalized color histogram of the reference image 202.Hereupon, in the normalized color histogram, a value that the number ofthe pixels having the same gradation divided by the number of the totalpixels is used as the degree of each gradation, and the sum the degreesof every gradations is normalized to be “1”.

When the numbers of the image data of the input image 201 and thereference image 202 are small, the color histograms of them will be thediscrete comb shape including the gradation of zero degree, as shown inFIGS. 30C and 30D. Furthermore, when the luminance in the input image201 and/or the reference image 202 are/is varied or when the blushingoccurs in one or both of the images, the color histogram of the inputimage 201 will be discrepant from that of the reference image 202, asshown in FIG. 30E, so that the similarity between the input image 201and the reference image 202 becomes much lower. Thus, an area to beretrieved will erroneously be judged as the area not including thereference image. The similarity is a value calculated that the number ofdegrees in the color histograms of the input image 201 and the referenceimage 202 are compared with respect to each gradation, and the smallerdegrees are added with respect to every gradations.

Another example that the number of the image data of the input image 211is largely different from that of the reference image 212 is describedwith reference to FIGS. 31A to 31G. FIG. 31A shows an input image 211.FIG. 31B shows a reference image 212. Numeral 213 in FIG. 31A designatesan area to be retrieved, and numeral 214 designates another area not tobe retrieved. FIG. 31C shows a normalized color histogram 215 of thearea 213. FIG. 31D shows a normalized color histogram 216 of the area214. FIG. 31E shows a normalized color histogram 217 of the referenceimage 212.

In this example, the number of the image data of the area 213 is smallerthan that of the area 214, but the number of the image data of thereference image 212 is similar to that of the area 214.

As can be seen from FIG. 31C, the color histogram 215 which is formed bybasing the small number of the image data has a discrete comb shapeincluding the gradation of zero degree. On the other hand, as can beseen from FIGS. 31D and 30E, the color histograms 216 and 217 which areformed by basing the relatively large number of the image datarespectively have successive curves taking positive values.

FIG. 31F shows the color histograms 215 and 217 which are superimposedon the same coordinates. In FIG. 31F, hatched portions 218 correspond tothe similarity of the color histogram 215 of the area 213 and the colorhistogram 217 of the reference image 212. FIG. 31G shows the colorhistograms 216 and 217 which are superimposed on the same coordinates.In FIG. 31G, a hatched portion 219 corresponds to the similarity of thecolor histogram 216 of the area 214 and the color histogram 217 of thereference image 212.

As can be seen from FIGS. 31F and 31G, the color histogram 215 of thearea 213 has the comb shape, so that the similarity of the hatchedportions 218 is smaller than that of the hatched portion 219 withrespect to the color histogram 217. Thus, the area 214 which is not tobe retrieved will erroneously be retrieved as the area including thereference image instead of the area 213 to be retrieved.

As mentioned above, when the color histogram becomes the comb shape, thesimilarity of the histogram of an area of the input image with respectto that of the reference image becomes lower even though the gradationis discrepant a little. Especially, when the luminance in the inputimage is varied, the image retrieving performance will become muchlower. Furthermore, when the number of the image data of the input imageor the reference image is largely different from the number of the imagedata of the area to be compared, the image retrieving performance willbe reduced.

SUMMERY OF THE INVENTION

An object of the present invention is to provide an image retrievingapparatus and a method executed therein, by which an area similar to areference image can quickly be retrieved in an input image withoutomission.

Another object of the present invention is to provide an imageretrieving apparatus and a method executed therein, by which an areasimilar to a reference image can precisely be retrieved in an inputimage when a number of image data is small.

An image retrieving apparatus in accordance with the present inventionretrieves whether an image similar to a predetermined retrieving imageto be retrieved is included in an input image or not by comprising thefollowing elements.

A first area extracting unit extracts a first retrieving area having afirst size from the input image with respect to each movement at a firstmoving pitch. A first histogram forming unit forms a first histogramwith respect to each first retrieving area with a first resolution ofgradation. A second histogram forming unit forms a second histogram ofthe retrieving image with the first resolution of gradation. A secondarea extracting unit compares the first histogram with the secondhistogram for calculating a similarity of the first histogram withrespect to the second histogram, and extracts a retrieving area havingthe similarity larger than a first level. A third area extracting unitextracts a second retrieving area having a second size from the firstretrieving area extracted by the second area extracting unit at a secondmoving pitch. A third histogram forming unit forms a third histogramwith respect to each second retrieving area with a second resolution ofgradation which is higher than the first resolution of gradation. Afourth histogram forming unit forms a fourth histogram of the retrievingimage with the second resolution of gradation. An area retrieving unitcompares the third histogram with the fourth histogram for calculating asimilarity of the third histogram with respect to the fourth histogram,and retrieves an area having the similarity larger than a second level.

By such a configuration, the image retrieving of the retrieving image inthe input image is executed at two stages with different sizes ofretrieving areas and different resolutions of gradation of the colorhistograms. At first, at least one candidate area in which theretrieving image can be included is extracted in the input image by arough image retrieving with a larger size of the retrieving areas and arough (lower) resolution of gradation of the color histograms.Subsequently, an area in which the retrieving image is included isextracted in the candidate area by a fine image retrieving with asmaller size of the retrieving areas and a fine (higher) resolution ofgradation of the color histograms. Thus, it is possible to lighten theburden for calculating the histograms and to shorten the time for theimage retrieving process.

Another image retrieving apparatus in accordance with the presentinvention retrieves whether an image similar to a predeterminedretrieving image to be retrieved is included in an input image or not bycomprising the following elements.

An area extracting unit extracts a retrieving area having apredetermined size from the input image with respect to each movement ata predetermined moving pitch. A judging unit judges whether a number ofpixels included in the retrieving area is smaller than a predeterminedvalue or not. A first histogram forming unit forms a first histogramwith respect to each retrieving area with a first resolution ofgradation, and smoothes the first histogram when the number of pixels inthe retrieving area is smaller than the predetermined value. A secondhistogram forming unit forms a smoothed second histogram of theretrieving image. An area retrieving unit calculates a similarity of thefirst histogram of each retrieving area with respect to the secondhistogram by comparing the first histogram with the second histogram,and retrieves an area having the similarity larger than a predeterminedlevel.

By such a configuration, the histograms are smoothed with having no combshape, so that the similarity of the histogram of the retrieving area inthe input image with respect to that of the retrieving image cannot belower due to the discrepant between the gradations. Thus, it is possibleto retrieve the area in the input image similar to the retrieving imageprecisely.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an image retrieving apparatus in accordancewith a first embodiment of the present invention;

FIG. 2 is a drawing showing an input image and a retrieving area movingin the input image;

FIG. 3 is a graph for showing an example of a three dimensional colorhistogram;

FIGS. 4A and 4B are graphs for showing examples of the color histogramsrespectively formed with the resolution of gradations N=16 and N=256;

FIGS. 5A to 5C are graphs for showing a method for calculating asimilarity between the color histograms;

FIG. 6A is a drawing for showing a relation between the input image andthe retrieving area moving in the input image in rough image retrievingin the first embodiment;

FIG. 6B is a drawing for showing the retrieving image 13 to be retrievedin the rough image retrieving in the first embodiment;

FIGS. 6C to 6E are graphs for showing a method for calculating asimilarity between the color histograms in the rough image retrieving inthe first embodiment;

FIG. 7A is a drawing for showing relations between a candidate area tobe retrieved and the input image and between the retrieving area and thecandidate area moving in the fine image retrieving in the firstembodiment;

FIG. 7B is a drawing for showing the retrieving image to be retrieved inthe fine image retrieving in the first embodiment;

FIGS. 7C to 7E are graphs for showing a method for calculating asimilarity between the color histograms in the fine image retrieving inthe first embodiment;

FIG. 8 is a drawing for showing an example of the candidate areaobtained by the rough image retrieving in the first embodiment;

FIGS. 9A to 9F are drawings respectively for show relations between theinput image and the retrieving area or the like in the image retrievingprocess in the first embodiment;

FIG. 10 is a flowchart for showing a main routine of the imageretrieving steps in the first embodiment;

FIG. 11 is a flowchart for showing a subroutine of the rough imageretrieving in step #110 in the main routine shown in FIG. 10;

FIG. 12 is a flowchart for showing a subroutine of the fine imageretrieving in step #115 in the main routine shown in FIG. 10;

FIG. 13 is a flowchart for showing a subroutine for deciding anobjective area in step #305 in the subroutine shown in FIG. 12;

FIG. 14 is a drawing for showing an example that the candidate areas arecontinued and/or overlapped;

FIG. 15 is a drawing for showing an example that a retrieving image isdisposed for bridging two retrieving areas;

FIG. 16 is a block diagram for showing an example of an electricconfiguration of a digital still camera using the image retrievingapparatus in accordance with the first embodiment;

FIG. 17 is an HQ chromaticity diagram for showing an area in which acolor of human skin can be reproduced properly;

FIGS. 18A to 18C are drawings respectively for showing examples offilters used in edge emphasizing process in a digital still camera usingthe image retrieving apparatus in accordance with the first embodiment;

FIG. 19 is a graph for showing examples of gradation characteristics (γcharacteristic curves) used in gradation compensation process in thedigital still camera in accordance with the first embodiment;

FIG. 20 is a block diagram for showing an example of an electricconfiguration of a printer using the image retrieving apparatus inaccordance with the first embodiment

FIG. 21 is a block diagram of an image retrieving apparatus inaccordance with a second embodiment of the present invention;

FIGS. 22A to 22E are drawings respectively for show relations between aninput image and a retrieving area or the like in image retrievingprocess in the second embodiment;

FIGS. 23A to 23D are graphs for showing smoothing process of a colorhistogram in the second embodiment;

FIG. 24 is a flowchart for showing a main routine of image retrievingsteps in the second embodiment;

FIG. 25 is a flowchart for showing a subroutine for forming a normalizedcolor histogram in steps #515 and #525 in the main routine shown in FIG.24;

FIG. 26 is a flowchart for showing a main routine of image retrievingsteps in a modification of the second embodiment;

FIG. 27 is a flowchart for showing a subroutine for selecting resolutionof gradation “N” in steps #720 in the main routine shown in FIG. 26;

FIGS. 28A to 28E are graphs for showing smoothing process of a colorhistogram in another modification of the second embodiment;

FIG. 29 is a graph for showing examples of gradation characteristics (γcharacteristic curves) used in gradation compensation process in thedigital still camera in accordance with the second embodiment;

FIG. 30A is the drawing for showing the relation between the input imageand the retrieving area moving in the input image in the conventionalimage retrieving method;

FIG. 30B is the drawing for showing the retrieving image to be retrievedin the conventional image retrieving method;

FIGS. 30C to 30E are the graphs for showing the problem in theconventional method for calculating the similarity between the colorhistograms;

FIG. 31A is the drawing for showing the relation between the input imageand the retrieving image in the conventional image retrieving method;

FIG. 31B is a drawing for showing the reference image used in theconventional image retrieving method; and

FIGS. 31C to 31G are graphs of the color histograms for showing theconventional image retrieving method.

DETAILED DESCRIPTION OF THE EMBODIMENT

First Embodiment

A first embodiment of the present invention is described with referenceto the drawings.

FIG. 1 shows a block diagram of an image retrieving apparatus inaccordance with the first embodiment. The image retrieving apparatus 10comprises an image input unit 1, a color converter 2, a retrieving areasetting unit 3, an HQ histogram forming unit 4, an HQ histogramcomparator 5, a similarity judging unit 6, an area position memory 7,and a similar area information output unit 8.

The image retrieving apparatus 10 retrieves whether an input image 11(see FIG. 2) includes a retrieving image similar to a reference image ofan object to be retrieved or not by comparing the similarity of colorhistograms of the image in an area of the input image and the retrievingimage. In this description, a part of the input image similar to thereference image is abbreviated as “retrieving image”.

At first, an area in which the retrieving image can be existed isroughly retrieved. Subsequently, the position of the retrieving image isprecisely retrieved in the roughly retrieved area. In the rough imageretrieving of the retrieving image, color histograms having lowresolution of gradation are used. In the fine retrieving of theretrieving image, color histograms having high resolution of gradationare used. In both of the rough image retrieving and the fine imageretrieving, a size of an area to be compared is selected and the area ismoved in predetermined directions at a predetermined pitch so as to scanwhole of the input image or the roughly retrieved area. A colorhistogram is calculated with respect to the image portion at each stopposition of the area. All the color histograms are compared with thecolor histogram of the reference image. In the first embodiment, a humanface portion is used as the retrieving image to be retrieved as shown inFIG. 2.

The image input unit 1 takes an input image 11 and a retrieving image 13(see FIG. 6B). For example, the input image 11 has 640 (horizontaldirection)×480 (vertical direction) pixels, and the retrieving image 13has 60×80 pixels. The input image 11 and the retrieving image 13 arerespectively taken as an image data configured by R(red), G(green) andB(blue) color signals.

The color converter 2 converts the image data configured by R(red),G(green) and B(blue) color signals to another image data configured byhue (H) and compensated saturation (Q) by following equations.

 H=cos⁻¹[{(R−G)+(R−B)}/2·1/√{square root over ()}{(R−G)²+(R−B)·(G−B)}]  (1)Q=[{(2R−G−B)/2}²+{√{square root over ( )}(3)(G−B)/2}²]  (2)

Since the hue (H) may not be affected by variation of luminance, it iseffective for retrieving an object in which the luminance of the objectwill be predicted. Since the compensated saturation (Q) has acharacteristic that the saturation value increases in proportion to thebrightness, it is preferable for detecting human skin having relativelyhigh brightness. It can emphasize the human skin much more than thesaturation obtained from the Munsell color system.

The retrieving area setting unit 3 sets the size of the retrieving area12 which is to be compared with the retrieving image in the rough imageretrieving and the fine image retrieving. In the first embodiment, theretrieving area 12 to be compared has a rectangular shape. The functionof the retrieving area setting unit 3 in the rough image retrieving isdescribed with reference to FIG. 2.

As shown in FIG. 2, a length of a horizontal side of the roughretrieving area 12 is set to be ph1, and a length of a vertical sidethereof is set to be pv1. When lengths of a horizontal and verticalsides of the input image 11 is shown as PH1 and PV1, the lengths ph1 andpv1 respectively shown by the following formulae.ph1≈PH1/4 and pv1≈PV1/4

By selecting the size of the rough retrieving area 12 is selected to be{fraction (1/16)} of the size of the input image 11, it is possible toexclude the areas in which the retrieving image is rarely included fromcandidate areas for the fine image retrieving. When the size of therough retrieving area 12 is much larger than a preferable size, there isa possibility to exclude the area in which the retrieving image isincluded. Alternatively, when the size of the rough retrieving area 12is much smaller than a preferable size, the quantity to be calculatedbecomes larger. In the first embodiment, the input image 11 has thepixels of 640×480, so that the rough retrieving area 12 has the pixelsof 160×120.

The pitch of the movement of the rough retrieving area 12 in thehorizontal direction is selected to be kh1 and that in the verticaldirection is selected to be kv1. The rough retrieving area 12 is movedby the pitch kh1 in the horizontal direction and by the pitch kv1 in thevertical direction so as to scan whole the input image 11. In FIG. 2,the rough retrieving area 12 illustrated by the solid line is moved bythe pitch kh1 in the horizontal direction from the position illustratedby the dotted line.

In the first embodiment, the pitches kh1 and kv1 in the movement of therough retrieving area 12 is selected askh1≈ph1/2 and kv1≈pv1/2.

Thus, the pitch kh1 corresponds to 80 pixels and the pitch kv1corresponds to 60 pixels.

Subsequently, the function of the retrieving area setting unit 3 in thefine image retrieving is described. When a position of the roughretrieving area to be precisely retrieved is decided by referring theposition information of the candidate areas memorized in the areaposition memory 7. The fine image retrieving is executed so that a fineretrieving area is set in the decided rough retrieving area.

In the first embodiment, the fine retrieving area 15 (see FIG. 7A) isset to be a rectangular shape, and the coordinates (shi, svi) (i=1, 2, 3. . . ) of the pixel at the upper left end of the area is used as theinformation with respect to the position of the fine retrieving area.

A size “p” of the fine retrieving area 15 shown in FIG. 7A is set to bethat a length of a horizontal side and a length of a vertical side ofthe fine retrieving area 15 are initially set to be ph2 and pv2. Whenthe fine image retrieving with respect to the rough retrieving area 12is completed by using the fine retrieving area 15 having a predeterminedsize “p”, the size “p” of the fine retrieving area 15 is reduced by adownsizing ratio “r”. The fine image retrieving will be repeated by thesame manner until the size “p” of the fine retrieving area 15 becomesequal to or smaller than a predetermined size “P”. In the firstembodiment, the initial values of the lengths of the horizontal andvertical sides ph2 and pv2 are selected asph2=ph1 andpv2=pv1.

The downsizing ratio “r” is selected to be r=0.8. By repeating the imageretrieving with the reduction of the size of the fine retrieving area15, it is possible to prevent the missing of the retrieving with norelation to the size of the retrieving image.

The predetermined size “P” is selected to be {fraction (1/10)} of thesize of the input image 11. Thus, the pixels of the predetermined size“P” becomes 64×48 pixels. This size is selected to be the minimum sizein view of the case that a human face portion is existed as a part of anobject in the input image.

The pitch of the movement of the fine retrieving area 15 in thehorizontal direction is selected to be kh2 and that in the verticaldirection is selected to be kv2. The fine retrieving area 15 is moved bythe pitch kh2 in the horizontal direction and by the pitch kv2 in thevertical direction so as to scan whole the input image 11. Since thevalues of the pitches have the relations that kv1>kv2 and kh1>kh2, theimage retrieving in the fine image retrieving can precisely executedthan that in the rough image retrieving. In the first embodiment, thevalues of the pitches kh2 and kv2 are respectively set to be “1” as theminimum pitch of the movement. By such the selection, it is possible toexecute the image retrieving with no missing.

The HQ histogram forming unit 4 shown in FIG. 1 generates the normalizedcolor histograms with using the H and Q data of the input image withrespect to each retrieving area set by the retrieving area setting unit3. Furthermore, the HQ histogram forming unit 4 generates the normalizedcolor histograms with using the H and Q data of retrieving image.

The color histogram shows the number of pixels as the degrees, in whichthe number of pixels in a predetermined area is two-dimensionallycounted with respect to each of the hue (H) and the compensatedsaturation (Q). The color histogram becomes three-dimensional as shownin FIG. 3 which shows an example of the shape of the color histogram. InFIG. 3, the coordinates corresponding to the hue (H) and the compensatessaturation (Q) respectively having the largest number of the pixels inthe retrieving area take the largest values. The hue (H) takes a valuebetween 0 to 2π (0° to 360°), and the compensated saturation (Q) takes avalue between 0 to the largest value among the values of R, G and B.

The normalized color histogram is the color histogram normalized thatthe sum of the degrees is to be “1” by dividing the number of pixelswith respect to each gradation by the total number of the pixels in theretrieving area.

The HQ histogram forming unit 4 varies the resolution of gradation forforming the color histograms in the rough image retrieving and the fineimage retrieving. When the resolution of gradation in the rough imageretrieving is shown by a symbol “Na” and the resolution of gradation inthe fine image retrieving is shown by a symbol “Nb”, the resolution “Na”is selected to be smaller than the resolution “Nb”. In the firstembodiment, the resolution “Na” is selected to be 16 and the resolution“Nb” is to be 256.

The resolution of gradation is the finesse of the gradation. Theresolution of gradation “N” means that the number of the division of thegradation is to be “N”. When the resolution of gradation is selected tobe “N” and the total number of the gradation is to be 256, a width ofthe gradation becomes 256/N. The color histogram having a largeresolution of gradation “N” is called “fine resolution color histogram”and the histogram having a small resolution of gradation “N” is called“rough resolution color histogram”.

In the first embodiment, the resolution of gradation “Nb” in the fineimage retrieving is selected to be 256 which corresponds to the highestresolution of gradation of the image retrieving apparatus in accordancewith the first embodiment. The resolution of gradation “Na” in the roughimage retrieving is selected to be smaller than the highest resolutionof gradation of the image retrieving apparatus. In the first embodiment,the resolution of gradation “Na” in the rough image retrieving isselected to be 16 which is proper to show the color distribution by thehistogram. It, however, is possible to select another value such as 32as the resolution of gradation “Na” in the rough image retrieving.

FIGS. 4A and 4B show examples of the color histograms which are formedfrom the same image data having the 256 gradations. FIG. 4A shows thecolor histogram having the resolution of gradation N=16. FIG. 4B showsthe color histogram having the resolution of gradation N=256. Forsimplifying the explanation, the color histograms are abbreviated to beone-dimensional.

Since the resolution of gradation “Na” in the rough image retrieving ismade smaller, the number of the gradation to be compared for calculatingthe similarity can be reduced. In the example shown in FIGS. 4A and 4B,the number of the gradation becomes {fraction (1/16)}. Thus, the imageretrieving can be made faster owing to the shortening of the calculationtime.

The HQ histogram comparator 5 show in FIG. 1 compares the colorhistogram of the retrieving area of the input image with the colorhistogram of the retrieving image. The similarity judging unit 6calculates the similarity “S” between the compared color histograms andjudges whether the similarity “S” is higher than a predetermined levelor not. In the rough image retrieving, the similarity “S” is comparedwith a predetermined first level “S1”. In the fine image retrieving, thesimilarity “S” is compared with a predetermined second level “S2” whichis larger than the first level “S1”.

In the first embodiment, the first level “S1” is selected to be 0.5(S1=0.5) which is a relatively low value. By such the selection, it ispossible to retrieving an area including a human face portion as acandidate portion, even when the size of the human face portion issmaller than the retrieving area and the value of the similarity “S”becomes smaller. Furthermore, the second level “S2” is selected to be0.8 (S2=0.8). By such the selection, the image retrieving can beexecuted more precisely in the fine image retrieving.

A method for calculating the similarity between the color histograms isdescribed with reference to FIGS. 5A to 5C, 6A to 6E and 7A to 7E. FIG.5A shows a normalized color histogram 21 which is formed from aretrieving area of the input image. FIG. 5B shows another normalizedcolor histogram 22 formed from a retrieving image. FIG. 5C shows thatthe color histograms 21 and 22 are compared. FIGS. 6A to 6E shows stepsfor forming the color histogram and for judging the similarity in therough image retrieving. FIGS. 7A to 7E shows steps for forming the colorhistogram and for judging the similarity in the fine image retrieving.In these figures, the histograms are abbreviated as one-dimensional.

For calculating the similarity “S”, the degrees of the normalized colorhistograms 21 and 22 shown in FIGS. 5A and 5B are compared with respectto respective gradations, and the values of the degrees with respect torespective gradations are summed. Thus, the sum of the degrees in ahatched portion 23 with respect to respective gradation in FIG. 5Ccorresponds to the similarity “S” (0≦S≦1). The larger the similarity “S”becomes, the larger the degrees of the coincidence of both imagesbecome.

FIG. 6A shows a relation between an input image 11 and a retrieving area12. FIG. 6B shows a retrieving image 13. In the first embodiment, theretrieving image 13 is a human face portion.

FIGS. 6C and 6D respectively show a normalized color histogram 31 of theretrieving area 12 and a normalized color histogram 32 of the retrievingimage 13 which are formed by the HQ histogram forming unit 4. In therough image retrieving, the resolution of gradation “Na” is selected tobe relatively smaller (for example, Na=16).

The color histograms 31 and 32 shown in FIGS. 6C and 6D are compared bythe HQ histogram comparator 5, and a hatched portion 33 shown in FIG. 6Eis obtained. A similarity “S” is calculated by the similarity judgingunit 6 based on the hatched portion 33. Subsequently, the similarity “S”is compared with the first level “S1” and a candidate area is decided bythe result of the comparison.

FIG. 7A shows relations between the input image 11 and a candidate area14 which is obtained by the rough image retrieving, and between thecandidate area 14 and a retrieving area 15 (hatched portion). FIG. 7Bshows the retrieving image 13.

FIGS. 7C and 7D respectively show a normalized color histogram 34 of theretrieving area 15 and a normalized color histogram 35 of the retrievingimage 13 which are formed by the HQ histogram forming unit 4. In thefine image retrieving, the resolution of gradation “Nb” is selected tobe relatively larger (for example, Nb=256).

The color histograms 34 and 35 shown in FIGS. 7C and 7D are compared bythe HQ histogram comparator 5, and a hatched portion 36 shown in FIG. 7Eis obtained. A similarity “S” is calculated by the similarity judgingunit 6 based on the hatched portion 36. Subsequently, the similarity “S”is compared with the second level “S2” and an area in which theretrieving image 13 is included is decided by the result of thecomparison.

The area position memory 7 in FIG. 1 memorizes positions of theretrieving areas which have the similarities “S” larger than the firstlevel “S1” or the second level “S2”. In the rough image retrieving, theretrieving areas having the similarity “S” larger than the first level“S1” are memorized as the candidate areas which will be to be retrievedby the fine image retrieving. In the fine image retrieving, theretrieving area having the largest similarity “S” is memorized as thearea in which the retrieving image 13 includes.

FIG. 8 shown an example of the candidate area obtained by the roughimage retrieving. In this example, “N” number of the candidate areas areexisted in the input image 11. A first candidate area A1 is memorized inthe area position memory 7 as size information ph1 and pv1, and aposition information of a coordinate (h1, v1) at the upper left endthereof. An N-th candidate area AN is memorized in the area positionmemory 7 as size information ph1 and pv1, and a position information ofa coordinate (hN, vN) at the upper left end thereof.

The similar area information output unit 8 shown in FIG. 1 outputs thearea including the retrieving image 13 memorized in the area positionmemory 7 as a result of the image retrieving.

Subsequently, steps of the image retrieving in the image retrievingapparatus in accordance with the first embodiment is described withreference to FIGS. 9A to 9F and 10. FIGS. 9A to 9F respectively showrelations between the input image 11 and the retrieving area 12 or thelike. FIG. 10 is a flowchart showing a main routine of the imageretrieving steps.

In the step #100 in FIG. 10, the input image 11 and the retrieving image13 to be retrieved are taken as the image data based on the R, G and Bsignals (see FIGS. 9A and 9B). Subsequently, the image data based on theR, G and B signals are converted to other image data based on the H andQ data (#105).

In the step #110, the rough image retrieving for obtaining candidateareas 14 in which the similarity “S” between the color histograms of theretrieving area 12 of the input image 11 and the retrieving image 13 ishigher than the first level “S1” is executed (see FIGS. 9C and 9D).Details of the rough image retrieving will be described below withreference to FIG. 11 showing a subroutine flow.

Subsequently, in the step #115, the fine image retrieving for obtainingan area 16 including the retrieving image 13 by basing on the similarity“S” between the color histograms of the retrieving area 15 in thecandidate area 14 and the retrieving image 13 is higher than the secondlevel “S2” is executed (see FIGS. 9E and 9F). Details of the fine imageretrieving will be described below with reference to FIG. 12 showing asubroutine flow.

In FIG. 11 showing the subroutine of the rough image retrieving in thestep #110, a normalized color histogram of the retrieving image 13 isformed with the resolution of gradation Na=16 (#200). Subsequently, anormalized color histogram of the retrieving area 12 of the input image11 is formed with the resolution of gradation Na=16 (#205).

A similarity “Sm” between the normalized color histograms is calculated(#210), and the similarity “Sm” is compared with the first level “S1”(#215). When the similarity “Sm” is larger than the first level “S1”(Sm>S1: YES in the step #215), the position information with respect tothe retrieving area 12 is memorized in the area position memory 7(#220).

When the similarity “Sm” is equal to or smaller than the first level“S1” (Sm≦S1: NO in the step #215) or when the position information ismemorized in the step #220, it is judged whether the movement of theretrieving area 12 is scanned whole the input image 11 or not (#225).When the whole of the input image 11 has not been scanned (NO in thestep #225), the retrieving area 12 is moved by the predetermined pitchkv1 in the vertical direction or kh1 in the horizontal direction (#230)and returns to the step #205. Alternatively, when the whole of the inputimage 11 has been scanned (YES in the step #225), the retrieving areas12 memorized in the area position memory 7 in the step #220 are selectedas the candidate areas 14 (#235), and this subroutine will be completed.

In FIG. 12 showing the subroutine of the fine image retrieving in thestep #115, a normalized color histogram of the retrieving image 13 isformed with the resolution of gradation Nb=256 (#300). Subsequently, anarea to which the fine image retrieving is decided among the candidateareas 14 (#305). Details of the steps for deciding an objective area tobe precisely retrieved will be described with reference to FIG. 13showing a subroutine flow thereof.

A normalized color histogram of a retrieving area 15 in the objectivearea is formed with the resolution of gradation Nb=256 (#310).Subsequently, a similarity “Sn” between the normalized color histogramsis calculated (#315), and the similarity “Sn” is compared with thesecond level “S2” (#320). When the similarity “Sn” is larger than thesecond level “S2” (Sn>S2: YES in the step #320), the positioninformation with respect to the retrieving area 15 is memorized in thearea position memory 7 (#325).

When the similarity “Sn” is equal to or smaller than the second level“S2” (Sn≦S2: NO in the step #320) or when the position information ismemorized in the step #325, it is judged whether the movement of theretrieving area 15 is scanned whole the objective area or not (#330).When the whole of the objective area has not been scanned (NO in thestep #330), the retrieving area 15 is moved by the predetermined pitchkv2 in the vertical direction or kh2 in the horizontal direction (#335)and returns to the step #310.

When the whole of the objective area has been scanned (YES in the step#330), a size “p” of the retrieving area 15 is compared with apredetermined size “P” (#340). When the size “p” of the retrieving area15 is larger than the predetermined size “P” (p>P: YES in the step#340), the size “p” of the retrieving area 15 is downsized by thedownsizing ratio “r” (#345), and returns to the step #310.Alternatively, when the size “p” of the retrieving area 15 is equal toor smaller than the predetermined size “P” (p≦P: NO in the step #340),the position information of the retrieving area 15 memorized in the areaposition memory 7 in the step #325 is outputted as the retrieving result(#350), and this subroutine flow will be completed.

In FIG. 13 showing the subroutine for deciding the objective area in thestep #305, it is judged whether N number of the candidate areasretrieved by the rough image retrieving are continued or overlapped bybasing on the coordinates (hi, vi) (i=1 to N) and the sizes (ph1, pv1)with respect to respective candidate areas (#400).

With respect to the independent candidate areas which are judged not tobe continued or overlapped (NO in the step #400), each candidate area isjudged as the objective area to be retrieved precisely (#405). In thiscase, the coordinate (shi, svi) of each objective area corresponds tothe coordinate of each independent candidate area.

On the other hand, with respect to the candidate areas which are judgedto be continued or overlapped (YES in the step #400), a rectangular areaenclosing the continued or overlapped areas is selected as the objectivearea to be retrieved precisely (#410). Subsequently, an initial size ofthe retrieving area in the objective area having a length ph2 of ahorizontal side and a length pv2 of a vertical side is selected (#415),and this subroutine flow will be completed.

FIG. 14 shows an example that the candidate areas are continued and/oroverlapped. When the candidate areas are continued and/or overlapped,the position information of an candidate area is compared with theposition information of another candidate area. The smallest vales ofthe coordinates (hi, vi) of the candidate areas are designated byhi_(min) and vi_(min), and the largest values of them are designated byhi_(max) and vi_(max).

In the above-mentioned case, the coordinate (shi, svi) of the objectivearea is shown as (shi, svi)=(hi_(min), vi_(min)). The size of theobjective area (a length ph2 of the horizontal side and a length pv2 ofthe vertical side thereof) is shown asph2 =hi _(max)+ph1 −hi _(min), andpv2 =vi _(max)+pv1 −vi _(min).

It is assumed that the candidate areas 141, 142 and 143 shown in FIG. 14are continued and/or overlapped. The coordinates of the candidate areas141, 142 and 143 are respectively shown as (h1, v1), (h2, v2) and (h3,v3). The lengths of the horizontal and vertical sides of them arecommonly to be ph1 and pv1. At this time, the minimum values of hi andvi correspond to h1 and v1. The maximum values of hi and vi correspondto h2 and v3. Thus, the coordinate the objective area 140 illustrated bydotted line in the figure becomes (h1, v1). The lengths ph2 and pv2 ofthe horizontal side and the vertical side of it will beph2=h2+ph1−h1, andpv2=v3+pv1−v1.

In the above-mentioned first embodiment, the candidate areas 14 having apossibility that the retrieving image 13 is included are retrieved atfirst by executing the rough image retrieving with using colorhistograms 31 and 32 having low resolution of gradation “Na”.Subsequently, the area including the retrieving image 13 is retrieved byexecuting the fine image retrieving with using the color histograms 34and 35 having high resolution of gradation “Nb” in the candidate areas14. Since the resolution of gradation “Na” used in the rough imageretrieving is lower, it is possible to reduce the burden of thecalculation in the rough image retrieving and to shorten the timenecessary for the image retrieving. Furthermore, the fine imageretrieving is executed with respect to only the candidate areas 14obtained in the rough image retrieving. Thus, it is possible to reducethe total burden of the calculation and to shorten the time necessaryfor retrieving the image.

The pitches kh2 and kv2 of the movement of the retrieving area 15 in thefine image retrieving are respectively selected to be one pixel, so thatit is possible to prevent the missing of the image retrieving. Thepitches kh1 and kv1 of the movement of the retrieving area 12 in therough image retrieving are respectively selected to be larger than thepitches kh2 and kv2 (kh1>kh2, kv1>kv2), so that it is possible toshorten the time necessary for the rough image retrieving.

In the rough image retrieving, since the first level “S1” serving as athreshold value for judging the similarity “S” is selected to berelatively small value such as 0.5 (S1=0.5), it is possible to preventthe missing of the image retrieving. In the fine image retrieving, sincethe second level “S2” is selected to be relatively large value such as0.8 (S2=0.8>S1), it is possible to execute the fine image retrievingwith a high accuracy. Furthermore, since the second level “S2” isselected not to be so large value such as 0.9 or 0.95, it is possible toretrieve not only the same image as the retrieving image 13 but also theimage similar to the retrieving image such as a face of another person.Thus, the image retrieving method in accordance with the firstembodiment can be applied for processing the most suitable imageprocessing to a human image among the input images.

Modifications of the first embodiment will be described below. In thefirst embodiment, the pitches kh1 and kv1 of the movement of theretrieving area 12 in the rough image retrieving are selected to bekh1≈ph1/2 and kv1≈pv1/2. The pitches kh1 and kv1 in the rough imageretrieving are not restricted by the above-mentioned example. Forexample, when the pitches kh1 and kv1 in the rough image retrieving areselected to be kh1≈ph1 and kv1≈pv1 which are substantially the same asthe lengths of the sides of the retrieving area 12, there is apossibility that the retrieving image 13 disposed for bridging theretrieving areas 12 as shown in FIG. 15 cannot be retrieved. Thus, it ispreferable to select the pitches kh1 and kv1 of the movement of theretrieving image in the rough image retrieving smaller than half valuesof the lengths of the horizontal side and the vertical side of theretrieving area 12.

The coordinate at the upper left end of the retrieving area 12 or 15 ismemorized as the position information of the candidate area or the areaincluding the retrieving image. It, however, is possible to memorize thecoordinate at the center of the retrieving area 12 or 15 as the positioninformation.

The first level “S1” and the second level “S2” serving as the thresholdvalues for judging the similarity “S” are respectively selected to beS1=0.5 and S2=0.8. The values of the first level “S1” and the secondlevel “S2” can be varied corresponding to the desired accuracy of theimage retrieving.

In the above-mentioned first embodiment, the hue (H) and the compensatedsaturation (Q) are used as the color space. It, however, is possible touse the R, G and B signals. Furthermore, it is possible to use anothercolor system such as the HIS (Hue, Intensity, Saturation) color system,the L*a*b* color system, or the L*u*v* color system.

It is possible further to provide an operating unit 91 illustrated bydotted line in FIG. 1 showing the configuration of the image retrievingapparatus in accordance with the first embodiment. By such amodification, it is possible to input the values of the parameters suchas the values of the resolution of gradation “Na” and “Nb”, the sizes ofthe retrieving areas 12 and 15, the values of the pitches kh1, kh2, kv1and kv2, the values of the first level “S1” and the second level “S2”,and so on by using the operating unit 91.

In the above-mentioned first embodiment, the retrieving image 13 istaken by the image input unit 1. It, however, is possible further toprovide an retrieving data memory 92 illustrated by dotted line in FIG.1. The data with respect to the retrieving image 13 is previouslymemorized in the retrieving data memory 92. In this modification, it ispossible to memorize the R, G and B signals as the data of theretrieving image 13. Alternatively, it is possible to memorize the H andQ data converted from the R, G and B signals as the data of theretrieving image 13.

Furthermore, it is possible to memorize the normalized color histogrambased on the H and Q data as the data of the retrieving image 13 in theretrieving data memory 92. In this case, it is further possible tomemorize the normalized color histograms which are formed with both ofthe resolution of gradation “Na” and “Nb”. Alternatively, it is possibleto memorize the normalized color histogram with the resolution ofgradation “Nb” only. The normalized color histogram with the resolutionof gradation “Na” is calculated from the normalized color histogram withthe resolution of gradation “Nb”.

FIG. 16 shows a block diagram for showing an example of an electricconfiguration of a digital still camera using the image retrievingapparatus in accordance with the first embodiment.

An imaging unit 101 of the digital still camera 100 includes an areaimaging device such as CCD, in which a plurality of photo-electroconverting elements are arranged in two-dimensional, a set of colorfilters are disposed in front of each photo-electro converting elements.The imaging unit 101 converts optical energy corresponding to an imageof an object 109 to electrical color image signals 101R, 101G and 101Bcorresponding to the color filters and outputs the color image signals101R, 101G and 101B.

An optical lens system 102 includes a taking lens, an aperture and adriving mechanism for moving the taking lens and the aperture. Theoptical lens system 102 focuses the image of the object 109 on thesurface of the imaging device of the imaging unit 101.

An image retrieving apparatus 10 corresponds to that shown in FIG. 1.The imaging retrieving apparatus 10 retrieves whether an input imagecorresponding to the color image signals 101R, 101G and 101B includes ahuman face portion (the object 109 includes a human face portion) or notprior to a shutter switch in an operation unit 107 is switched on.

An imaging operation controller 103 controls the driving mechanism ofthe optical lens system 102 by following a control program memorized ina memory unit 104. The imaging operation controller 103 executed anautomatic focusing control for focusing the focus of the taking lens ofthe optical lens system 102 on the human face portion of the objectretrieved by the image retrieving apparatus 10.

Furthermore, the imaging operation controller 103 executes an automaticexposure control for driving the driving mechanism of the optical lenssystem 102 and the imaging unit 101 so as to take a predeterminedaperture value and a predetermined shutter speed (or exposing time) bywhich the human face portion becomes a proper exposure value.

In this example, the proper exposure value is EV±0 with respect to aproper exposure value corresponding to the sensitivity of the imagingdevice. When the exposure value is designated by eight bit data (0 to255), a mean value of luminance Y in the retrieving area satisfies100≦Y≦150.

The mean value of luminance Y can be obtained by the following equation,when the values of the color imaging signals 101R, 101G and 101B arerespectively designated by symbols “R”, “G” and “B”.Y=0.299R+0.587G+0.114B

An image processing unit 105 executes predetermined image processing tothe color image signals 101R, 101G and 101B by following a controlprogram memorized in the memory unit 104. The image processing unit 105executes an automatic white balance control for adjusting the ratio ofthe output of the color image signals 101R and 101B with respect to thecolor image signal 101G so that the color data corresponding to thehuman face portion is included in a proper area 108 as shown in FIG. 17.

FIG. 17 is an HQ chromaticity diagram for showing an area in which acolor of human skin can be reproduced properly. On the HQ chromaticitydiagram, a direction toward 0° corresponds red (R), a direction toward+120° corresponds green (G) and a direction toward +240° (−120°)corresponds blue (B). For example, a color data at a point “P” can bedesignated by hue (H) which is an angle from 0°, and compensatedsaturation (Q) which is a distance from the center of the chromaticitydiagram.

In the example shown in FIG. 17, the proper area 108 is enclosed by30°≦H≦60° and 40≦Q≦150. Since the digitalized value is shown by eightbit data (0 to 255), so that the compensated saturation Q takes a valuebetween 0 to 255.

The image processing unit 105 adjusts the ratio of the output of thecolor image signals 101R and 101B with respect to the color image signal101G so that the color data corresponding to the human face portion isincluded in a proper area 108 as shown in FIG. 17, when the color imagesignals 101R, 101G and 101B are converted to the H and Q data byfollowing the above-mentioned equations (1) and (2).

It is possible to memorize the proper area 108 in the memory unit 104,previously. Alternatively, it is possible to input the proper area 108by using the operation unit 107.

Furthermore, image processing unit 105 varies a degree for edgeemphasizing operation with respect to the retrieving area equal to orsmaller than a predetermined level, when the human face portion isretrieved in the input image by the image retrieving apparatus 10. Atthis time, the degree for edge emphasizing operation is reducedcorresponding to the size of the retrieving area including the humanface portion or the size of the human face portion.

Table 1 shows the degree for the edge emphasizing operation and thegradation characteristic (γ) with respect to each region of the size ofthe human face portion. FIGS. 18A to 18C respectively show examples offilters used in the edge emphasizing operation. FIG. 18A shows thefilter having a high degree of edge emphasizing effect. FIG. 18B showsthe filter having a middle degree of edge emphasizing effect. FIG. 18Cshows the filter having a low degree of edge emphasizing effect.

TABLE 1 RATIO OF DEGREE OF HUMAN FACE EDGE GRADATION PORTION EMPHASIZINGCHARACTERISTIC LARGE (30 to 100%) WEAK γ = 0.4 (a in FIG. 19) MIDDLE (10to 30%) MIDDLE γ = 0.45 (b in FIG. 19) SMALL (5 to 10%) MIDDLE γ = 0.5(c in FIG. 19) NOT RETRIEVED STRONG γ = 0.55 (d in FIG. 19)

As can be seen from table 1, when the ratio of the size of theretrieving area including the human face portion with respect to thesize of the input image is in the range from 30 to 100%, the filtershown in FIG. 18C having the low degree for edge emphasizing effect isused for emphasizing the edge of the retrieving area including the humanface portion. When the ratio of the size of the retrieving areaincluding the human face portion with respect to the size of the inputimage is in the range from 5 to 30%, the filter shown in FIG. 18B havingthe middle degree for edge emphasizing effect is used for emphasizingthe edge of the retrieving area. On the other hand, when the human faceportion is not retrieved in the input image, the filter shown in FIG.18A having the high degree for edge emphasizing effect is used foremphasizing the edge of the retrieving area.

Still furthermore, the image processing unit 105 varies the gradationcompensation process with respect to whole the input image correspondingto the size of the retrieving area including the human face portion orthe size of the human face portion.

FIG. 19 shows examples of the gradation characteristics (γcharacteristic curves) used in the gradation compensation process by theimage processing unit 105.

As can be seen from table 1, when the ratio of the size of theretrieving area including the human face portion with respect to thesize of the input image is in the range from 30 to 100%, the γcharacteristic curve “a” (γ=0.4) shown in FIG. 19 is used forcompensating the gradation. When the ratio of the size of the retrievingarea including the human face portion with respect to the size of theinput image is in the range from 10 to 30%, the γ characteristic curve“b” (γ=0.45) shown in FIG. 19 is used for compensating the gradation.When the ratio of the size of the retrieving area including the humanface portion with respect to the size of the input image is in the rangefrom 5 to 10%, the γ characteristic curve “c” (γ=0.5) shown in FIG. 19is used for compensating the gradation. On the other hand, when thehuman face portion is not retrieved in the input image, the γcharacteristic curve “d” (γ=0.55) shown in FIG. 19 is used forcompensating the gradation.

A boundary, for example, 30% of the regions of the ratio of the size ofthe retrieving area with respect to the size of the input image are tobe included in one of the adjoining two regions. The boundaries are notrestricted by the examples shown in table 1. It is possible to selectproper values corresponding to the characteristic of the digital stillcamera 100.

The image data of the object 109 after the image processing by the imageprocessing unit 105 is memorized in the memory unit 104 or displayed ona display unit 106. The memory unit 104 is, for example, configured by aROM, a RAM, an EEPROM or the like. The display unit 106 is configuredby, for example, an LCD.

The above-mentioned modification is described with respect to thedigital still camera. The image retrieving apparatus 10 in accordancewith the first embodiment can be applied to another imaging apparatussuch as a digital video camera for recording a movie.

FIG. 20 shows a block diagram for showing an example of an electricconfiguration of a printer using the image retrieving apparatus inaccordance with the first embodiment.

A data receiving unit 111 of the printer 110 receives an image databased on the R, G and B signals transmitted from, for example, apersonal computer (PC), and outputs the color image signals 111R, 111Gand 111B.

An image retrieving apparatus 10 corresponds to that shown in FIG. 1.The imaging retrieving apparatus 10 retrieves whether an input imagecorresponding to the color image signals 101R, 101G and 101B includes ahuman face portion or not.

An image processing unit 112 executes image processing operation to thecolor image signals 111R, 111G and 111B so as to print an imageincluding a human portion properly by following a control programmemorized in a memory unit 113. The image processing unit 112 adjuststhe ratio of the output of the color image signals 111R, 111G and 101Bso that the luminance corresponding to the human face portion becomesproper.

In this example, the proper value of the luminance is defined that amean value of luminance Y in the retrieving area satisfies 100≦Y≦150,when the luminance value is designated by eight bit data (0 to 255).

The mean value of luminance Y can be obtained by the following equation,when the values of the color imaging signals 111R, 111G and 111B arerespectively designated by symbols “R”, “G” and “B”.Y=0.299R+0.587G+0.114B

The image processing unit 112 further executes a color balanceprocessing for adjusting the ratio of the output of the color imagesignals 111R, 111G and 111B so that the color data corresponding to thehuman face portion is included in a proper area 108 as shown in FIG. 17.By such the color balance processing, the human face portion can beprinted with a proper color.

It is possible to memorize the proper area 108 in the memory unit 113,previously. Alternatively, it is possible to input the proper area 108by using the operation unit 115.

Furthermore, image processing unit 112 varies a degree for edgeemphasizing operation with respect to the retrieving area correspondingto the size of the retrieving area including the human face portion withrespect to the size of the input image, similar to the above-mentioneddigital still camera 100. By such the image processing, it is possibleto restrict the sharpness of the human face portion so as not to be muchhigher, so that the human face portion can be printed properly.

Still furthermore, the image processing unit 112 varies the gradationcompensation process with respect to whole the input image correspondingto the size of the retrieving area including the human face portion orthe size of the human face portion as shown in FIG. 19, similar to theabove-mentioned digital still camera 100. By such the image processing,it is possible to restrict the gradation of the human face portion so asnot to be much higher, so that the human face portion can be printedwith proper gradation.

The image data processed by the image processing unit 112 is used by aprinting unit 114 for printing the image on a paper sheet.

Second Embodiment

A second embodiment of the present invention is described with referenceto the drawings.

FIG. 21 shows a block diagram of an image retrieving apparatus inaccordance with the second embodiment. The image retrieving apparatus 10comprises an image input unit 1, a color converter 2, a retrieving areasetting unit 3, an HQ histogram forming unit 4, an HQ histogramcomparator 5, a similarity judging unit 6, an area position memory 7,and a similar area information output unit 8.

In comparison with FIGS. 1 and 21, it is found that the image retrievingapparatus 10 in the second embodiment is very similar to that in thefirst embodiment, so that the explanation of the common elements areomitted.

The image retrieving apparatus 10 retrieves whether an input image 11(see FIG. 22A) includes a retrieving image 13 (see FIG. 22B) similar toa retrieving image of an object to be retrieved or not by comparing thecolor histograms of the image in an area of the input image and theretrieving image.

In the image retrieving by the image retrieving apparatus 10 inaccordance with the second embodiment, a retrieving area 12 having anoptional size is selected in the input image 11 as shown in FIG. 22C.The retrieving area 12 is moved in predetermined directions at apredetermined pitch so as to scan whole the input image 11. A colorhistogram of an image in each retrieving area 12 is compared with acolor histogram of the retrieving image 13. In the second embodiment, itis possible to retrieve several sizes of the retrieving image 13 byvarying the size of the retrieving area 12.

When the number of pixels in the retrieving area 12 or the retrievingimage 13 is equal to or smaller than a predetermined value due to thepixel density of the input image 11 is smaller or the size of the inputimage 11 is smaller, the image retrieving apparatus 10 generatessmoothed color histograms in order to prevent the reduction of the imageretrieving performance. In the second embodiment, a human face portionis used as the retrieving image to be retrieved as shown in FIG. 22B.

The image input unit 1 takes an input image 11 and a retrieving image13. For example, the input image 11 has 640 (horizontal direction)×480(vertical direction) pixels, and the retrieving image 13 has 15×15pixels. The input image 11 and the retrieving image 13 are respectivelytaken as an eight bit image data configured by R(red), G(green) andB(blue) color signals.

The color converter 2 converts the image data configured by R(red),G(green) and B(blue) color signals to another image data configured byhue (H) and compensated saturation (Q) by the above-mentioned equations(1) and (2) in the first embodiment.

The retrieving area setting unit 3 sets the size of the retrieving area12 which is to be compared with the retrieving image 13. In the secondembodiment, the retrieving area 12 has a rectangular shape, and aninitial size “p” of the retrieving area 12 is selected to be ⅘ of thesize of the input image 11. The retrieving area setting unit 3 furthermoves the retrieving area 12 by a predetermined pitch in eachpredetermined direction. The image retrieving is executed after eachmovement of the retrieving area 12.

When the image retrieving of the whole input image 11 by moving theretrieving area 12 is completed, the size “p” of the retrieving area 12is reduced by a downsizing ratio “r”, as shown in FIG. 22D. The imageretrieving will be repeated by the same manner until the size “p” of theretrieving area 12 becomes equal to or smaller than a predetermined size“P”.

In the second embodiment, the initial size “p” of the retrieving area 12is selected in view of the maximum size of the human face portion whichcan be included in the input image 11. By selecting the initial size “p”of the retrieving area 12 be ⅘ of the size of the input image 11, it ispossible to prevent to missing the retrieving of the largest human faceportion which can be existed in the input image 11. Since the number ofthe pixels of the input image 11 is 640×480 pixels, the initial value ofthe pixels of the retrieving area 12 becomes 512×384 pixels.

Since the downsizing ratio “r” is selected to be r=0.8 in the secondembodiment, the size “p” of the retrieving area 12 will be downsized tobe 410×307, 328×246, 262×197, . . . pixels. The number of the pixels arerounded to be the integer. By repeating the image retrieving with thereduction of the size of the retrieving area 12, it is possible toprevent the missing of the retrieving with no relation to the size ofthe retrieving image 13.

The predetermined size “P” is selected to be {fraction (1/10)} of thesize of the input image 11. Thus, the pixels of the predetermined size“P” becomes 64×48 pixels. This size is selected to be the minimum sizein view of the case that a human face portion is existed as a part of anobject in the input image 11.

The pitches of the movement of the retrieving area 12 in both directionare respectively set to be “1” as the minimum pitch of the movement. Bysuch the selection, it is possible to execute the image retrieving withno missing. The position of the retrieving area 12 is designated byusing any one of the coordinates at the four corners and the center ofthe rectangular.

The HQ histogram forming unit 4 shown in FIG. 21 generates thenormalized color histograms with using the H and Q data of the inputimage with respect to each retrieving area set by the retrieving areasetting unit 3. Furthermore, the HQ histogram forming unit 4 generatesthe normalized color histograms with using the H and Q data ofretrieving image.

The HQ histogram forming unit 4 generates a color histogram with apredetermined resolution of gradation “N”. As shown in FIGS. 4A and 4Bshowing the examples of the color histograms respectively having theresolution of gradation N=16 and N=256, it is found that the imageretrieving can be made faster owing to the shortening of the calculationtime by reducing the resolution of gradation “N”. In the secondembodiment, the resolution of gradation “N” is selected to be 256(N=256) corresponding to the highest resolution of image retrievingapparatus 10.

The HQ histogram forming unit 4 further judges whether the number ofpixels of the retrieving area 12 and the retrieving image 13 is smallerthan a predetermined value “D” or not. When the number of pixels issmaller than the predetermined value “D”, the HQ histogram forming unit4 generates a smoothed color histogram in which the degrees of thehistogram are smoothed. In the second embodiment, the predeterminedvalue “D” is selected to be 256 corresponding to the highest resolutionof image retrieving apparatus 10.

The smoothening process of the degree is described with reference toFIGS. 23A to 23D. In the smoothing process, the interpolation isexecuted at two steps. At first, the interpolation of the degree isexecuted with respect to a gradation having a positive value of thedegree but extremely smaller than the degrees of the neighboringgradations. Subsequently, the interpolation of the degree is executedwith respect to a gradation having the value zero of the degree.

As shown in FIGS. 23A and 23B, with respect to a predetermined gradation“ni” having a positive value “Pi” of degree, other gradations “nk” and“nj” respectively having positive values “Pk” and “Pj” of degrees areconsidered in the higher gradation side and the lower gradation side.The smaller one of the values “Pk” and “Pj” is selected to be theminimum degree P_(min). When a ratio of the value “Pi” of the gradation“ni” with respect to the minimum degree P_(min) is smaller than apredetermined ration, for example, Pi≦P_(min)/3, it is judged that thevalue “Pi” of degree of the gradation “ni” is extremely smaller than thevalues “Pk” and “Pj” of the neighboring gradations “nk” and “nj”. Thus,the value “Pi” of degree of the gradation “ni” is converted by thefollowing equation (3).Pi=Pj+(Pk−Pj)·(ni−nj)/(nk−nj)  (3)

By such the converting process, the value “Pi” of degree of thegradation “ni” will be changed to be a value shown by dotted line inFIG. 23B from the original value shown by solid line in FIG. 23A. Thevalue “Pi” after the conversion corresponds to a linearly interpolatedvalue of the values “Pk” and “Pj”.

Subsequently, as shown in FIGS. 23C and 23D, with respect to apredetermined gradation “nd” having a value zero of degree, othergradations “ni” and “nj” respectively having positive values “Pi” and“Pj” of degrees are considered in the higher gradation side and thelower gradation side. The value “Pd” of degree of the gradation “nd” isconverted by the following equation (4).Pd=Pj+(Pi−Pj)·(nd−nj)/(ni−nj)  (4)

By such the converting process, the value “Pd” of degree of thegradation “nd” will be changed to be a value shown by dotted line inFIG. 23D from the original value zero shown in FIG. 23A. The value “Pd”after the conversion corresponds to a linearly interpolated value of thevalues “Pi” and “Pj”.

The HQ histogram forming unit 4 further executes the normalization ofthe color histograms. The normalized color histogram is the colorhistogram normalized that the sum of the degrees is to be “1” bydividing the number of pixels with respect to each gradation by thetotal number of the pixels in the retrieving area.

The HQ histogram comparator 5 show in FIG. 21 compares the colorhistogram of the retrieving area 12 of the input image 11 with the colorhistogram of the retrieving image 13. The similarity judging unit 6calculates the similarity “Sm” between the compared color histograms andjudges whether the similarity “Sm” is higher than a predetermined level“S” or not. A retrieving area 12 having the similarity “Sm” larger thanthe predetermined level “S” is judged as the area in which theretrieving image 13 is included. The predetermined level “S” can beselected to be a suitable value corresponding to the desired imageretrieving accuracy. In the second embodiment, the predetermined level“S” is selected to be 0.8 (S=0.8). The value of the similarity “S” canbe obtained by the same manner shown in FIGS. 5A to 5C in the firstembodiment.

The area position memory 7 in FIG. 21 memorizes positions of theretrieving areas 12 which have the similarities “Sm” larger than thepredetermined level “S” as an area 14 in which the retrieving image 13is included (see FIG. 22E). The similar area information output unit 8outputs the area 14 including the retrieving image 13 memorized in thearea position memory 7 as a result of the image retrieving.

Subsequently, steps of the image retrieving in the image retrievingapparatus in accordance with the second embodiment is described withreference to FIG. 24. FIG. 24 is a flowchart showing a main routine ofthe image retrieving steps.

In the step #500, the input image 11 and the retrieving image 13 to beretrieved are taken as the image data based on the R, G and B signals(see FIGS. 22A and 22B). Subsequently, the image data based on the R, Gand B signals are converted to other image data based on the H and Qdata (#505). The H and Q data of the retrieving image 13 is taken(#510), and the normalized color histogram of the retrieving image 13 isformed (#515). Details of the forming of the normalized color histogramwill be described below with reference to FIG. 25 showing a subroutineflow.

Subsequently, the H and Q data of the retrieving area 12 in the inputimage 11 (see FIG. 22C) is taken (#520), and the normalized colorhistogram of the retrieving area 12 is formed (#525).

A similarity “Sm” between the normalized color histograms is calculated(#530), and the similarity “Sm” is compared with the predetermined level“S” (#535). When the similarity “Sm” is larger than the predeterminedlevel “S” (Sm>S: YES in the step #535), the position information withrespect to the retrieving area 12 is memorized in the area positionmemory 7 (#540).

When the similarity “Sm” is equal to or smaller than the predeterminedlevel “S” (Sm≦S: NO in the step #535) or when the position informationis memorized in the step #540, it is judged whether the movement of theretrieving area 12 is scanned whole the input image 11 or not (#545).When the whole of the input image 11 has not been scanned (NO in thestep #545), the retrieving area 12 is moved by the predetermined pitchin the vertical or horizontal direction (#550) and returns to the step#520.

When the whole of the input image 11 has been scanned (YES in the step#545), the size “p” of the retrieving area 12 is compared with thepredetermined size “P” (#555). When the size “p” of the retrieving area12 is larger than the predetermined size “P” (p>P :NO in the step #555),the size “p” of the retrieving area 12 is downsized by the downsizingratio “r” as shown in FIG. 22D (#560), and returns to the step #520.Alternatively, when the size “p” of the retrieving area 12 is equal toor smaller than the predetermined size “P” (p≦P :YES in the step #555),the position information of the retrieving area 12 memorized in the areaposition memory 7 is outputted as the position information of the area14 in which the retrieving image 13 is included as shown in FIG. 22E(#565), and this subroutine flow is completed.

In FIG. 25 showing the subroutine for forming the normalized colorhistogram in the steps #515 and #525, the color histogram is formed withthe predetermined resolution of gradation “N” (N=256 in the secondembodiment) from the H and Q data of the retrieving image 13 or theretrieving area 12 of the input image 11 (#600). Subsequently, the totalnumber “Dt” of pixels of the image data is judged whether it is smallerthan a predetermined value “D” or not (#605).

When the total number “Dt” is smaller than the predetermined value “D”(YES in the step #605), the degrees of the gradations in the colorhistogram formed in the step #600 are judged whether the degree withrespect to each gradation is a positive value but equal to or smallerthan a predetermined value or not (#610). In the second embodiment, thepredetermined value is P_(min)/3 when a smaller value of degrees of theneighboring gradations in the high gradation side and the low gradationside is selected as the minimum value P_(min).

When the degree with respect to the gradation is the positive value butequal to or smaller than the predetermined value (YES in the step #610),the interpolation of the data is executed by following theabove-mentioned equation (3) (#615). When the degree with respect to thegradation is not the positive value and larger than the predeterminedvalue (NO in the step #610) or when the interpolation of the data iscompleted in the step #615, it is judged whether the judgment of thedegrees with respect to all the gradations has been completed or not(#620). When the judgment has not been completed, it will return to thestep #610, and the above-mentioned steps be repeated.

When the judgment of the degrees with respect to all the gradation hasbeen completed (YES in the step #620), the degree with respect to eachgradation of the color histogram is judged whether the value of thedegree is zero or not (#625). When the value of the degree is zero (YESin the step #625), the interpolation of the data is executed byfollowing the above-mentioned equation (4) (#630). When the value of thedegree is not zero (NO in the step #625) or when the data isinterpolated in the step #630, it is judged whether the judgment of thedegrees with respect to all the gradations has been completed or not(#635). When the judgment has not been completed, it will return to thestep #625, and the above-mentioned steps be repeated.

When the total number “Dt” is equal to or larger than the predeterminedvalue “D” (NO in the step #605) or when the judgment of the degrees withrespect to all the gradation has been completed (YES in the step #635),the color histogram is normalized (#640), and this subroutine flow willbe completed.

In the second embodiment, the input image 11 has 640×480 pixels, and theretrieving image 13 has 15×15=225 pixels, so that the total number 225of pixels of the retrieving image 13 is smaller than the predeterminedvalue D=256. Thus, the smoothed color histogram of the retrieving image13 can be formed.

As mentioned above, the total number of pixels of the image data isjudged whether it is equal to or smaller than the predetermined value“D” or not and the smoothing process is executed to the color histogramof the image data when the total number of pixels is equal to or smallerthan the predetermined value “D”. Thus, it is possible to prevent thecolor histogram of the image data be the comb shape. Furthermore, it ispossible to prevent the large reduction of the similarity between thehistograms due to a minute discrepancy of the gradation when thehistograms have comb shapes. Still furthermore, it is possible toprevent the reduction of the image retrieving performance when thenumber of pixels of the input data becomes much larger.

In the smoothing process, the interpolation of the vale of the degree isexecuted with respect to the value of the degree when it is a positivevalue but extremely smaller than other values of the degrees of theneighboring gradations. Subsequently, the interpolation of the value ofthe degree is executed with respect to the gradation having the value ofthe degree is zero. Thus, the histogram having a smoothed shape cansurely be formed, so that it is possible to prevent that the histogramhas a comb shape.

A modification of the image retrieving apparatus in accordance with thesecond embodiment will be described. The electrical block diagram of themodified image retrieving apparatus is substantially the same as thatshown in FIG. 21. The smoothing process by the HQ histogram forming unit4 is different.

In this modification, the HQ histogram forming unit 4 executes thesmoothing process by roughing the resolution of gradation “N” when thenumber of pixels of the retrieving area 12 of the input image 11 or theretrieving image 13 is equal to or smaller than the predetermined value“D”.

The HQ histogram forming unit 4 compares a number of pixels “Dn” of theretrieving area 12 with a number of pixels “Dk” of the retrieving image13. The HQ histogram forming unit 4 further compares the smaller value“K” of the numbers “Dn” and “Dk” with the predetermined value “D”. WhenK<D, it selects the resolution of gradation N=K/5 which will be used forforming a color histogram.

Steps of the image retrieving in the modified image retrieving apparatusin accordance with the second embodiment is described with reference toFIG. 26. FIG. 26 is a flowchart showing a main routine of the imageretrieving steps.

In the step #700, the input image 11 and the retrieving image 13 to beretrieved are taken as the image data based on the R, G and B signals(see FIGS. 22A and 22B). Subsequently, the image data based on the R, Gand B signals are converted to other image data based on the H and Qdata (#705).

The H and Q data of the retrieving image 13 is taken, and the number ofpixels “Dk” is counted (#710). The number of pixels “Dk” in thismodification is 15×15=255. Similarly, the H and Q data of the retrievingarea 12 is taken, and the number of pixels “Dn” is counted (#715). Sincethe initial value of the size of the retrieving area 12 is ⅘ of the sizeof the input image 11, the initial value of the number of pixels “Dn” ofthe retrieving area 12 becomes 512×384=196608.

Subsequently, the resolution of gradation “N” which will be used forforming the color histograms of the retrieving area 12 and theretrieving image 13 is selected (#720). Details of the selection of theresolution of gradation “N” will be described below with reference toFIG. 27 showing a subroutine flow. The normalized color histogram of theretrieving image 13 is formed with using the resolution of gradation “N”(#725).

Subsequently, the image data of the retrieving area 12 in the inputimage 11 (see FIG. 22C) is taken (#730), and the normalized colorhistogram of the retrieving area 12 is formed (#735).

A similarity “Sm” between the normalized color histograms is calculated(#740), and the similarity “Sm” is compared with the predetermined level“S” (#745). When the similarity “Sm” is larger than the predeterminedlevel “S” (Sm>S: YES in the step #745), the position information withrespect to the retrieving area 12 is memorized in the area positionmemory 7 (#750).

When the similarity “Sm” is equal to or smaller than the predeterminedlevel “S” (Sm≦S: NO in the step #745) or when the position informationis memorized in the step #750, it is judged whether the movement of theretrieving area 12 is scanned whole the input image 11 or not (#755).When the whole of the input image 11 has not been scanned (NO in thestep #755), the retrieving area 12 is moved by the predetermined pitchin the vertical or horizontal direction (#760) and returns to the step#730.

When the whole of the input image 11 has been scanned (YES in the step#755), the size “p” of the retrieving area 12 is compared with thepredetermined size “P” (#765). When the size “p” of the retrieving area12 is larger than the predetermined size “P” (p>P :NO in the step #765),the size “p” of the retrieving area 12 is downsized by the downsizingratio “r” as shown in FIG. 22D (#770), and returns to the step #715 soas to be counted the number of pixels “Dn” with respect to the downsizedsize of the retrieving area 12. Alternatively, when the size “p” of theretrieving area 12 is equal to or smaller than the predetermined size“P” (p≦P :YES in the step #765), the position information of theretrieving area 12 memorized in the area position memory 7 is outputtedas the position information of the area 14 in which the retrieving image13 is included as shown in FIG. 22E (#775), and this subroutine flow iscompleted.

In FIG. 27 showing the subroutine for selecting the resolution ofgradation “N” in the steps #720, the number of pixels “Dk” of theretrieving image 13 is compared with the number of gradation “Dn” of theretrieving area 12 (#800), and the smaller value of “Dk” and “Dn” isselected as the number of pixels “K” (#805 and #810).

Subsequently, the number of pixels “K” is compared with thepredetermined value “D” (#815). When the number of pixels “K” is smallerthan the predetermined value “D” (K<D) (YES in the step #815), the valueof the resolution of gradation “N” is selected to be K/5 (N=K/5) (#820).When the number of pixels “K” is equal to or larger than thepredetermined value “D” (K≧D) (NO in the step #815), the value of theresolution of gradation “N” is selected to be the maximum value of theresolution of gradation, for example 256 (#825).

In this modification, when the number of pixels Dk=15×15=255 and thesize “p” of the retrieving area 12 takes the initial value, the numberof pixels Dn=512×384=196608. Since the value “Dk” is smaller than thevalue “Dn” (Dk<Dn), the number of pixels “K” is selected to take thevalue “Dk” (K=Dk). Hereupon, there is a relation that K=225<D=256, sothat the number of the resolution of gradation “N” becomes 45(N=225/5=45). The color histograms are formed with the resolution ofgradation N=45.

In this modification, when the smaller number of pixels “k” of thenumber of pixels “Dn” of the retrieving area 12 and the number of pixels“Dk” of the retrieving image 13 is equal to or smaller than thepredetermined value “D”, the resolution of gradation “N” is elected tobe smaller such as N=K/5 used for forming the color histograms in thesmoothing process of the degrees. Thus, it is possible to prevent thatthe shape of the histogram becomes comb shape, and to prevent thereduction of the image retrieving performance.

Furthermore, the color histograms are formed by roughing the resolutionof gradation in the smoothing process of the degrees, so that the burdenin the calculation can be reduced and the time for retrieving the imagecan be shortened.

Hereupon, a width “n” of a gradation of the color histogram can beobtained by the following equation.n=(maximum resolution of gradation)/N

Furthermore, when it is supposed to occur the luminance variation orcolor fogging on the image of the object, it is preferable to increasethe width “n” of the gradation for reducing the affect of the variationof the luminance or the color fogging. For example, the width “n” of thegradation should be n=n+0.3 with respect to the hue (H) data, and thewidth “n” of the gradation should be n=n+30 with respect to thecompensates saturation (Q) data. The increased width +0.3 or +30 can bedecided by basing on the variation of the hue (H) or the compensatedsaturation (Q) caused by the color fogging or the under exposure on theimage taken by, for example, the digital still camera.

Another modification of the smoothing process of the degrees in theforming of the color histogram by the HQ histogram forming unit 4 isdescribed below with reference to FIGS. 28A to 28E.

FIG. 28A shows a basic color histogram having a comb shape due to thenumber of pixels is smaller. The numbers of the degrees with respect tothe gradations “n2” and “n5” are extremely smaller than the numbers ofdegrees of the neighboring gradations. The values of the degrees withrespect to the gradations “n3” and “n7” are zero.

FIG. 28B shows an example of an interpolated color histogram. Thepositive values of the degrees in the same histogram as shown in FIG.28A are serially bounded by dotted lines. This example, however, is notpreferable because the color histogram becomes a comb shape in thevicinity of the gradations “n2” and “n5” respectively having theextremely smaller values of the degrees.

FIG. 28C shows another example of an interpolated color histogram. Thepositive values of the degrees except the extremely smaller valuescorresponding to the gradations “n2” and “n5” in the same histogram asshown in FIG. 28A are serially bounded by solid lines. The value of thedegree with respect to the gradation “n2” is interpolated to be a valueon the solid line bounding the values of degrees with respect to thegradations “n1” and “n4” by following the above-mentioned equations (3)and (4). Similarly, the value of the degree with respect to thegradation “n5” is interpolated to be a value on the solid line boundingthe values of degrees with respect to the gradations “n4” and “n6”. Inthis example, the upper and lower limits of the gradation are linearlyinterpolated so that the predetermined minimum gradation such as zeroand the predetermined maximum gradation such as “255” becomes zero.Alternatively, it is possible to select the upper and lower limits ofthe gradation in a manner so that the gradation decided by basing on adifference between two gradations respectively taking positive values ofthe degrees on the higher limit side and the lower limit side should bezero.

FIG. 28D shows still another example of an interpolated color histogram.The values of the degree with respect to the gradations “n2” and “n5”are interpolated by substantially the same manner in the example shownin FIG. 28C. The values of degree with respect to the gradations “n3”and “n7” are interpolated to take the same value as the smaller one ofthe values with respect to adjoining gradations. By such theinterpolation, the color histogram can be formed with a relatively roughresolution of gradation corresponding to the number of the gradationshaving the positive values of degree.

FIG. 28E shows still another example of an interpolated color histogram.The values of the degree with respect to the gradations “n2” and “n5”which are extremely smaller and the values of degree with respect to thegradations “n3” and “n7” taking the value zero are interpolated to takethe same value as the smaller one of the values with respect toadjoining gradations. In this case the values of the degree with respectto the gradations “n2” and “n5” are regarded as zero. By such theinterpolation, the color histogram can be formed with a rough resolutionof gradation corresponding to the number of the gradations having thepositive values of degree except the gradations having extremely smallervalues and zero.

By using the smoothing process shown in FIG. 28C or 28D, it is possibleto prevent that the histogram have a comb shape even when the number ofpixels of the image data is smaller. Furthermore, when the smoothingprocess shown in FIG. 28E is used, the accuracy of the image retrievingis reduced than that in the case using the smoothing process shown inFIG. 28D, but it is possible to prevent that the color histogram has acomb shape. Especially, when the smoothing process shown in FIG. 28D or28E is used, the burden of the calculation can be reduced largely thanthe case using other smoothing process, so that the time of the imageretrieving can be shortened.

When the gradation having the extremely smaller value of degree such asthe gradation “n2” or “n5” in FIG. 28A is not existed in the histogram,it is possible to interpolate the values of degree by using the valueson the lined bounding the positive peak values of degree. By such theinterpolation, the histogram may not have a comb shape.

In the above-mentioned second embodiment, the hue (H) and thecompensated saturation (Q) are used as the color space. It, however, ispossible to use the R, G and B signals. Furthermore, it is possible touse another color system such as the HIS (Hue, Intensity, Saturation)color system, the L*a*b* color system, or the L*u*v* color system.

It is possible further to provide an operating unit 91 illustrated bydotted line in FIG. 21 showing the configuration of the image retrievingapparatus in accordance with the second embodiment. By such amodification, it is possible to input the values of the parameters suchas the value of the resolution of gradation “N”, the size “p” of theretrieving area 12, the values of the pitches of the retrieving area 12,the values of the predetermined level “S”, and so on by using theoperating unit 91.

In the above-mentioned second embodiment, the retrieving image 13 istaken by the image input unit 1. It, however, is possible further toprovide an retrieving data memory 92 illustrated by dotted line in FIG.21. The data with respect to the retrieving image 13 is previouslymemorized in the retrieving data memory 92. In this modification, it ispossible to memorize the R, G and B signals as the data of theretrieving image 13. Alternatively, it is possible to memorize the H andQ data converted from the R, G and B signals as the data of theretrieving image 13.

Furthermore, it is possible to memorize the normalized color histogrambased on the H and Q data as the data of the retrieving image 13 in theretrieving data memory 92. In this case, it is further possible tomemorize the normalized color histograms which are formed with theresolution of gradation “N”.

A digital still camera using the image retrieving apparatus inaccordance with the second embodiment is described. The digital stillcamera 100 has substantially the same configuration as shown in FIG. 16.The image retrieving apparatus 10 in the digital still camera 100corresponds to that shown in FIG. 21.

Table 2 shows the degree for the edge emphasizing operation and thegradation characteristic (γ) with respect to each region of the size ofthe human face portion. Filters used in the edge emphasizing operationare the same as them shown in FIGS. 18A to 18C.

TABLE 2 RATIO OF DEGREE OF HUMAN FACE EDGE GRADATION PORTION EMPHASIZINGCHARACTERISTIC LARGE (30 to 100%) WEAK γ = 1.1 (a in FIG. 29) MIDDLE (10to 30%) MIDDLE γ = 1.15 (b in FIG. 29) SMALL (5 to 10%) MIDDLE γ = 1.2(c in FIG. 29) NOT RETRIEVED STRONG γ = 1.25 (d in FIG. 29)

As can be seen from table 2, when the ratio of the size of theretrieving area including the human face portion with respect to thesize of the input image is in the range from 30 to 100%, the filtershown in FIG. 18C having the low degree for edge emphasizing effect isused for emphasizing the edge of the retrieving area including the humanface portion. When the ratio of the size of the retrieving areaincluding the human face portion with respect to the size of the inputimage is in the range from 5 to 30%, the filter shown in FIG. 18B havingthe middle degree for edge emphasizing effect is used for emphasizingthe edge of the retrieving area. On the other hand, when the human faceportion is not retrieved in the input image, the filter shown in FIG.18A having the high degree for edge emphasizing effect is used foremphasizing the edge of the retrieving area.

FIG. 29 shows examples of the gradation characteristics (γcharacteristic curves) used in the gradation compensation process by theimage processing unit 105. The gradation compensation process isexecuted corresponding to the size of the retrieving area after acompensation for reversing the input/output characteristics of amonitor.

As can be seen from table 2, when the ratio of the size of theretrieving area including the human face portion with respect to thesize of the input image is in the range from 30 to 100%, the γcharacteristic curve “a” (γ=1.1) shown in FIG. 29 is used forcompensating the gradation. When the ratio of the size of the retrievingarea including the human face portion with respect to the size of theinput image is in the range from 10 to 30%, the γ characteristic curve“b” (γ=1.15) shown in FIG. 29 is used for compensating the gradation.When the ratio of the size of the retrieving area including the humanface portion with respect to the size of the input image is in the rangefrom 5 to 10%, the γ characteristic curve “c” (γ=1.2) shown in FIG. 29is used for compensating the gradation. On the other hand, when thehuman face portion is not retrieved in the input image, the γcharacteristic curve “d” (γ=1.25) shown in FIG. 29 is used forcompensating the gradation.

A boundary, for example, 30% of the regions of the ratio of the size ofthe retrieving area with respect to the size of the input image are tobe included in one of the adjoining two regions. The boundaries are notrestricted by the examples shown in table 2. It is possible to selectproper values corresponding to the characteristic of the digital stillcamera 100.

The above-mentioned modification is described with respect to thedigital still camera. The image retrieving apparatus 10 in accordancewith the second embodiment can be applied to another imaging apparatussuch as a digital video camera for recording a movie.

Still furthermore, it is possible to apply the image retrievingapparatus 10 to a printer. A block diagram for showing an example of anelectric configuration of the printer using the image retrievingapparatus 10 in accordance with the second embodiment is substantiallythe same as that in the first embodiment shown in FIG. 20. Theexplanation of the printer is omitted.

Although the present invention has been fully described by way ofexample with reference to the accompanying drawings, it is to beunderstood that various changes and modifications will be apparent tothose skilled in the art. Therefore, unless otherwise such changes andmodifications depart from the scope of the present invention, theyshould be construed as being included therein.

1. An image retrieving apparatus for retrieving whether an image similarto a predetermined retrieving image to be retrieved is included in aninput image or not comprising: a first area extracting unit forextracting a first retrieving area having a first size from the inputimage with respect to each movement at a first moving pitch; a firsthistogram forming unit for forming a first histogram with respect toeach first retrieving area with a first resolution of gradation; asecond histogram forming unit for forming a second histogram of theretrieving image with the first resolution of gradation; a second areaextracting unit for comparing the first histogram with the secondhistogram for calculating a similarity of the first histogram withrespect to the second histogram and for extracting a retrieving areahaving the similarity larger than a first level; a third area extractingunit for extracting a second retrieving area having a second size fromthe first retrieving area extracted by the second area extracting unitat a second moving pitch; a third histogram forming unit for forming athird histogram with respect to each second retrieving area with asecond resolution of gradation which is higher than the first resolutionof gradation; a fourth histogram forming unit for forming a fourthhistogram of the retrieving image with the second resolution ofgradation; and an area retrieving unit for comparing the third histogramwith the fourth histogram for calculating a similarity of the thirdhistogram with respect to the fourth histogram and for retrieving anarea having the similarity larger than a second level.
 2. The imageretrieving apparatus in accordance with claim 1 further comprising amemory for memorizing at least the second histogram and the fourthhistogram which are previously formed by the second histogram formingunit and the fourth histogram forming unit.
 3. The image retrievingapparatus in accordance with claim 1 further comprising a memory formemorizing at least the fourth histogram which is previously formed bythe fourth histogram forming unit, and wherein the second histogramforming unit forms the second histogram from the fourth histogrammemorized in the memory.
 4. The image retrieving apparatus in accordancewith claim 1, wherein the first moving pitch is larger than the secondmoving pitch.
 5. The image retrieving apparatus in accordance with claim1, wherein the first level used in the second area extracting unit islower than the second level used in the area retrieving unit.
 6. Theimage retrieving apparatus in accordance with claim 1, wherein thesecond size is smaller than the first size.
 7. The image retrievingapparatus in accordance with claim 1, wherein the retrieving imageincludes a human face portion.
 8. The image retrieving apparatus inaccordance with claim 1, wherein the image retrieving apparatus is adigital still camera further comprising an imaging sensor for formingthe input image and an imaging controller for controlling an imagingprocess corresponding to a result of area retrieving by the arearetrieving unit.
 9. The image retrieving apparatus in accordance withclaim 8, wherein the imaging controller controls an optical system forfocusing the area retrieved by the area retrieving unit.
 10. The imageretrieving apparatus in accordance with claim 8, wherein the imagingcontroller controls an aperture value of an optical system and anexposing time for exposing the imaging sensor so as to be exposed thearea retrieved by the area retrieving unit with a proper exposingquantity.
 11. The image retrieving apparatus in accordance with claim 8,wherein the imaging controller adjusts color data with respect to thearea retrieved by the area retrieving unit.
 12. The image retrievingapparatus in accordance with claim 8, wherein the imaging controllerexecutes an edging process to an image data with respect to the arearetrieved by the area retrieving unit.
 13. The image retrievingapparatus in accordance with claim 8, wherein the imaging controllerexecutes a gradation compensation to an image data with respect to thearea retrieved by the area retrieving unit corresponding to the size ofthe area.
 14. The image retrieving apparatus in accordance with claim 1,wherein the image retrieving apparatus is a printer comprising animaging processor for executing imaging process to an image data withrespect to the area retrieved by the area retrieving unit, and aprinting unit for printing an image on a paper sheet with using theprocessed image data by the imaging processor.
 15. The image retrievingapparatus in accordance with claim 14, wherein the imaging processorcompensates output values of three principal color signals so as to makea luminance of an image with respect to the area retrieved by the arearetrieving unit be a proper value.
 16. The image retrieving apparatus inaccordance with claim 14, wherein the imaging processor adjusts colordata with respect to the area retrieved by the area retrieving unit. 17.The image retrieving apparatus in accordance with claim 14, wherein theimaging processor executes an edging process to an image data withrespect to the area retrieved by the area retrieving unit.
 18. The imageretrieving apparatus in accordance with claim 14, wherein the imagingprocessor executes a gradation compensation to an image data withrespect to the area retrieved by the area retrieving unit correspondingto the size of the area.
 19. An image retrieving method for retrievingwhether an image similar to a predetermined retrieving image to beretrieved is included in an input image or not comprising the steps of:a first area extracting step for extracting a first retrieving areahaving a first size from the input image with respect to each movementat a first moving pitch; a first histogram forming step for forming afirst histogram with respect to each first retrieving area with a firstresolution of gradation; a second histogram forming step for forming asecond histogram of the retrieving image with the first resolution ofgradation; a second area extracting step for comparing the firsthistogram with the second histogram for calculating a similarity of thefirst histogram with respect to the second histogram and for extractinga retrieving area having the similarity larger than a first level; athird area extracting step for extracting a second retrieving areahaving a second size from the first retrieving area extracted by thesecond area extracting step at a second moving pitch; a third histogramforming step for forming a third histogram with respect to each secondretrieving area with a second resolution of gradation which is higherthan the first resolution of gradation; a fourth histogram forming stepfor forming a fourth histogram of the retrieving image with the secondresolution of gradation; and an area retrieving step for comparing thethird histogram with the fourth histogram for calculating a similarityof the third histogram with respect to the fourth histogram and forretrieving an area having the similarity larger than a second level. 20.An image retrieving apparatus for retrieving whether an image similar toa predetermined retrieving image to be retrieved is included in an inputimage or not comprising: an area extracting unit for extracting aretrieving area having a predetermined size from the input image withrespect to each movement at a predetermined moving pitch; a judging unitfor judging whether a number of pixels included in the retrieving areais smaller than a predetermined value or not; a first histogram formingunit for forming a first histogram with respect to each retrieving areawith a first resolution of gradation, and for smoothing the firsthistogram when the number of pixels in the retrieving area is smallerthan the predetermined value; a second histogram forming unit forforming a smoothed second histogram of the retrieving image; and an arearetrieving unit for calculating a similarity of the first histogram ofeach retrieving area with respect to the second histogram by comparingthe first histogram with the second histogram, and for retrieving anarea having the similarity larger than a predetermined level.
 21. Theimage retrieving apparatus in accordance with claim 20, whereinsmoothing process in the first histogram forming unit and in the secondhistogram forming unit varies a value of degree of a specific gradationto be another value corresponding to a smaller one of values of degreeof neighboring gradations of the specific gradation in higher gradationside and lower gradation side.
 22. The image retrieving apparatus inaccordance with claim 21, wherein the smoothing process varies the valueof degree of the specific gradation in a manner so that a quantity ofdiscrepancy between the value of degree with respect to the specificgradation and one of the values of degree of neighboring gradations ofthe specific gradation in higher gradation side and lower gradation sidebecomes equal to or smaller than a predetermined value.
 23. The imageretrieving apparatus in accordance with claim 21, wherein the specificgradation is a gradation having a positive value of degree, a ratio ofthe value of degree of the specific gradation against a value of degreeof gradation which is the nearest positive value in the lower gradationside is equal to smaller than a predetermined value.
 24. The imageretrieving apparatus in accordance with claim 21, wherein the smoothingprocess varies the value of degree of the specific gradation to a valueon a straight line binding the values of degree of the neighboringgradations in the higher gradation side and the lower gradation side.25. The image retrieving apparatus in accordance with claim 21, whereinthe smoothing process further varies a value of degree of anothergradation taking zero degree to another value with using positive valuesof degree of the nearest gradations in the higher gradation side and thelower gradation side.
 26. The image retrieving apparatus in accordancewith claim 25, wherein the value of degree of another gradation takingzero degree is varied to a value on a straight line binding the positivevalues of degree of the nearest gradations in the higher gradation sideand the lower gradation side.
 27. The image retrieving apparatus inaccordance with claim 20, wherein the first histogram forming unit formsthe smoothed histogram with a second resolution of gradation lower thanthe first resolution of gradation.
 28. The image retrieving apparatus inaccordance with claim 27, wherein the second resolution of gradation isa value of the number of pixels in the retrieving area divided by apredetermined value.
 29. The image retrieving apparatus in accordancewith claim 20 further comprising a memory in which the second histogrampreviously formed by the second histogram forming unit is memorized. 30.The image retrieving apparatus in accordance with claim 20, wherein thejudging unit further judges whether a number of pixels of the retrievingimage is smaller than a predetermined value, and the second histogramforming unit forms the second smoothed histogram of the retrieving imagewhen the number of pixels is smaller than the predetermined value. 31.The image retrieving apparatus in accordance with claim 30, wherein thefirst histogram forming unit and the second histogram forming unitrespectively form the smoothed histograms with a resolution of gradationhaving a value which is divided a smaller one of the value of the numberof pixels of the retrieving area and the value of the number of pixelsof the retrieving image by a predetermined value when at least one ofthe number of pixels of the retrieving area and the number of pixels ofthe retrieving image is smaller than the predetermined value.
 32. Theimage retrieving apparatus in accordance with claim 20, wherein theretrieving image includes a human face portion.
 33. The image retrievingapparatus in accordance with claim 20, wherein the image retrievingapparatus is a digital still camera further comprising an imaging sensorfor forming the input image and an imaging controller for controlling animaging process corresponding to a result of area retrieving by the arearetrieving unit.
 34. The image retrieving apparatus in accordance withclaim 33, wherein the imaging controller controls an optical system forfocusing the area retrieved by the area retrieving unit.
 35. The imageretrieving apparatus in accordance with claim 33, wherein the imagingcontroller controls an aperture value of an optical system and anexposing time for exposing the imaging sensor so as to be exposed thearea retrieved by the area retrieving unit with a proper exposingquantity.
 36. The image retrieving apparatus in accordance with claim33, wherein the imaging controller adjusts color data with respect tothe area retrieved by the area retrieving unit.
 37. The image retrievingapparatus in accordance with claim 33, wherein the imaging controllerexecutes an edging process to an image data with respect to the arearetrieved by the area retrieving unit.
 38. The image retrievingapparatus in accordance with claim 33, wherein the imaging controllerexecutes a gradation compensation to an image data with respect to thearea retrieved by the area retrieving unit corresponding to the size ofthe area.
 39. The image retrieving apparatus in accordance with claim20, wherein the image retrieving apparatus is a printer comprising animaging processor for executing imaging process to an image data withrespect to the area retrieved by the area retrieving unit, and aprinting unit for printing an image on a paper sheet with using theprocessed image data by the imaging processor.
 40. The image retrievingapparatus in accordance with claim 39, wherein the imaging processorcompensates output values of three principal color signals so as to makea luminance of an image with respect to the area retrieved by the arearetrieving unit be a proper value.
 41. The image retrieving apparatus inaccordance with claim 39, wherein the imaging processor adjusts colordata with respect to the area retrieved by the area retrieving unit. 42.The image retrieving apparatus in accordance with claim 39, wherein theimaging processor executes an edging process to an image data withrespect to the area retrieved by the area retrieving unit.
 43. The imageretrieving apparatus in accordance with claim 39, wherein the imagingprocessor executes a gradation compensation to an image data withrespect to the area retrieved by the area retrieving unit correspondingto the size of the area.
 44. An image retrieving apparatus forretrieving whether an image similar to a predetermined retrieving imageto be retrieved is included in an input image or not comprising thesteps of: an area extracting step for extracting a retrieving areahaving a predetermined size from the input image with respect to eachmovement at a predetermined moving pitch; a judging step for judgingwhether a number of pixels included in the retrieving area is smallerthan a predetermined value or not; a first histogram forming step forforming a first histogram with respect to each retrieving area with afirst resolution of gradation, and for smoothing the first histogramwhen the number of pixels in the retrieving area is smaller than thepredetermined value; a second histogram forming step for forming asmoothed second histogram of the retrieving image; and an arearetrieving step for calculating a similarity of the first histogram ofeach retrieving area with respect to the second histogram by comparingthe first histogram with the second histogram, and for retrieving anarea having the similarity larger than a predetermined level.